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Understanding why students participate in multiple surveys: who are the hard-core 

responders? 

Abstract 

What causes a student to participate in a survey? This paper looks at survey response 
across multiple surveys to understand who the hard-core survey responders and non-responders 
are. Students at a selective liberal arts college were administered four different surveys 
throughout the 2002-2003 academic year, and we use the number of surveys participated in to 
understand how student characteristics such as demographics and personality affect cooperation. 
We find large altitudinal and behavioral differences between responders and non-responders. 
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Understanding why students participate in multiple surveys: who are the hard-core 

responders? 

Introduction 

What causes a student to participate in a survey? If we can better understand the answer 
to this question, we may be better able to develop techniques to counter survey nonresponse, 
which has been increasing in surveys of the general population (De Leeuw & Heer, 2002; T. W. 
Smith, 1995; Steeh, 1981) as well as in student surveys (Dey, 1997). Understanding the student 
decision to participate in a survey may also help us understand why some institutions are so 
successful in obtaining student participation in surveys, while others continually face very low 
response rates. For example, the National Survey of Student Engagement (NSSE) has response 
rates that vary across 316 different institutions from a low of 14% to a high of 70% (National 
Survey of Student Engagement, 2003). This variability is even more remarkable given that the 
survey instrument, sample composition (first-year students and seniors), method of 
administration, survey administrators (the NSSE center) and survey timing are all constant across 
these institutions. 

Most importantly, understanding nonresponse sheds much needed light on the quality of 
our survey data. Are the students who fill out our surveys “professional respondents” (Goyder, 
1986) who differ in significant ways from nonrespondents? If so, our surveys may 
systematically over- or under-estimate measures of engagement and satisfaction, as there may be 
a subset of students from whom we never hear. Given the increasing use of survey data for 
assessment, performance indicators, and measures of “best colleges,” it is vital that we 
understand exactly who fills out a survey and who does not. 
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There has been an increasing amount of research on student surveys, but the main focus 
has been on method of administration. We now know more about how paper versus web surveys 
(Kwak & Radler, 2002; Pealer, Weiler, Pigg Jr., Miller, & Dorman, 2001; Porter & Umbach, 
2004; Sax, Gilmartin, & Bryant, 2003), email surveys (Dommeyer & Moriarty, 2000), number of 
contacts (K. Smith & Bers, 1987), incentives (Porter & Whitcomb, 2003b; Zusman & Duby, 
1987), survey burden (Adams & Gale, 1982; Couper, Traugott, & Lamias, 2001; Crawford, 
Couper, & Lamias, 2001; Porter, Whitcomb, & Weitzer, 2004), and variations in the email 
contact (Porter & Whitcomb, 2003a; Whitcomb & Porter, 2004) all affect student survey 
response rates. Much less is known about how student characteristics, behaviors and attitudes 
affect survey response. 

In part this is due to the difficulty of collecting information about characteristics and 
attitudes of nonrespondents. A common approach is a follow-up survey of nonrespondents in an 
initial survey, but as Goyder (1986, p. 28) states. 

The epistemological limitation to surveys on surveys is self-evident; employing 
an instrument to measure its own performance is immediately contradictory. To 
assess attitudes toward the survey via attitude surveys is not unlike seeking to 
comprehend the mechanics of the camera solely from photographs. 

The issue here is simple: no matter what efforts are made to elicit cooperation from survey 
nonrespondents in a follow-up survey, large numbers of these nonrespondents will also refuse to 
participate in the follow-up survey. Thus any analyses will be biased, as these (twice) 
nonrespondents are likely to be quite different from respondents in the initial survey and follow- 



up survey. 
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We circumvent this problem in two ways. First, we use student database records and link 
these data to survey response, thus obtaining background data for all respondents and 
nonrespondents. Second, similar to Sax et al. (2003), we use a pre-college survey, the 
Cooperative Institutional Research Program first-year student survey (CIRP) administered by the 
Higher Education Research Institute (HERI), to obtain several behavioral and altitudinal 
measures. Because the CIRP has such a high response rate (90%-94% in this sample), it is more 
like a census than a survey, and thus avoids the issue of unrepresentativeness raised by 
nonresponse follow-up surveys. In addition, it is administered prior to the beginning of college 
and offers pre-college measures of behavior and attitudes, as well as valuable measures of 
previous survey refusal behavior. 

Our study is also unique because of our dependent variable. Previous research has 
looked at response and non-response in a single survey (e.g., Kuh, 2003; Sax et al., 2003), but 
“periennial” nonrespondents are a subset of nonrespondents in any given survey, as response is 
also greatly affected by salience (Goyder, 1982; Groves, Singer, & Corning, 2000; Heberlein & 
Baumgartner, 1978). Normally cooperative students may decline to participate in a single survey 
simply because the survey content bores them. In our research project, as part of a larger 
experiment a group of students were administered four different surveys throughout an academic 
year, and we use the number of surveys participated in to understand how student characteristics 
affect cooperation. By using multiple surveys rather than one, we are able to minimize the 
impact of salience on survey response. 

Einally, we focus not only on demographics but also student attitudinal data. Historically, 
researchers and practitioners have emphasized the relationship between demographic 
characteristics and survey response. One driving force behind this emphasis was a desire to 
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understand to what extent survey results capture the viewpoints of all campus constituencies. 
That is, demographic characteristics have been emphasized to ensure the representativeness of 
survey data across racial/ethnic and gender groups. By adding behavioral and altitudinal 
measures to the mix, this study can better estimate how survey results are affected by the error 
introduced by nonresponse. 



Literature review 

Methods for studying nonresponse 

In general there have been four main approaches to analyzing nonresponse. Figure 1 
presents stylized examples of these approaches. In each example, the base survey is conducted 
with a response rate of 40%; thus 40% of each bar in the “base survey” column is shaded to 
indicate the respondents. The bracket indicates the attempt to obtain additional data for members 
of the base sample. 

The first example estimates non-response bias by comparing individuals who respond 
early in the survey administration process to late responders. Guided by the continuum of 
resistance model (e.g., Lin & Schaeffer, 1995), this method assumes that late responders are very 
similar to non-responders. That is, had data collection ended at an earlier date, current late 
responders would be non-respondents. In the figure, 40% of the sample responds to the base 
survey, with one half of these individuals being classified as early responders. 

To understand the characteristics of survey non-respondents and the possible biases in 
survey data, time of response analyses tests for differences between early and late responders 
using the original survey as the source of information (e.g., Lahaut et ah, 2003; Voigt, Koepsell, 



& Baling, 2003). This method can be used to compares the demographic, as well as the 
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behavioral and attitudinal qualities of early and late responders, as long as this information was 
collected in the initial survey. One advantage of this method is the ability to estimate non- 
response bias with the survey instrument employed to collect data; additional data collection is 
not required. On the downside, however, the definition of a “late responders” is arbitrary and the 
cut point used by researchers may influence results. 

In the second example, nonresponse in a base survey is analyzed by conducting a follow- 
up survey of nonrespondents, usually with intense effort to ensure a high response rate (e.g., 
Brennan & Hoek, 1992; Kuh, 2003) \ The advantage of this approach is the ability to collect 
attitudinal and behavioral data for nonrespondents. But because the follow-up survey is focused 
on a non-cooperative group, these surveys never obtain information for all nonrespondents in the 
base survey. In the figure, 40% of the sample responds to the base survey, with a follow-up 
survey administered to the 60% who did not fill out the base survey. But only 20% of the 
original sample participates in the follow-up survey, meaning that we still know little or nothing 
about the 40% of the base sample who did not participate in either survey. 

A nonrespondent follow-up survey to the NSSE shows how difficult it can be to obtain 
the cooperation of nonrespondents. The NSSE nonrespondent survey obtained responses for less 
than a quarter of nonrespondents. Yet this low response rate was achieved even after shortening 
the original NSSE survey, using experienced telephone interviewers, and calling nonrespondents 
at least a dozen times (Kuh, 2003). Even with the most intense survey effort, a follow-up survey 
of nonrespondents is likely to result in a biased pool of respondents. 

* Note that follow-up surveys can take a more complicated form. Brennan and Hoek (1992), for example, surveyed 
both respondents and nonrespondents to their base survey, so their analysis was a cross between a simple 
nonresponse survey and a panel survey. Similarly, some panel surveys try to contact initial nonrespondents in later 
survey waves. 

^ Kuh (2003), p. 12, states that the NSSE follow-up survey contacted between 100 and 200 nonrespondents at 21 
schools, with 553 completed interviews. Thus the response rate lies between 13% (553/(200*21)) and 26% 
(553/(100*21)). 
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In the next example, a panel survey is used to understand nonresponse (e.g., Sax et al., 
2003; Singer, Groves, & Coming, 1999). Here respondents rather than nonrespondents are 
administered a second survey (and often additional waves of surveys). Nonresponse in later 
waves is analyzed, using data collected in the base survey. Similar to the previous example, one 
advantage of this approach is the ability to collect detailed data on members of the sample using 
the base survey. And while illuminating, these studies only aid understanding of nonresponse in 
the context of longitudinal surveys. In the figure, 60% of the sample who did not participate in 
the base survey is not examined. 

The fourth example shows the advantages of the record linkage approach, where records 
on all members of the base sample are linked to their survey response, and data from these 
records are used to understand nonresponse (e.g., Goyder, 1986, 1987; Goyder, Warriner, & 
Milller, 2002; Moore & Tamai, 2002). Goyder (1986), for example, combined survey data with 
municipal taxation records and city directories to obtain information on socioeconomic status 
(SES), age, and religion. By using non-survey data, this approach avoids the selection problem 
of nonresponse follow-up surveys and panel surveys. The disadvantage, however, is the lack of 
altitudinal and behavioral data. 

Correlates of nonresponse 

What do studies using these approaches tell us about survey nonresponse? We review the 
literature in three different areas to answer this question. The first area is studies of nonresponse 
in the general population. While this research does not analyze college students per se, the 
results are still informative about possible correlates of nonresponse among college students. 



The second area is studies of nonresponse among college students. These studies are more 
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informative in terms of our research topic, but unfortunately the amount of research in this area 
is limited. The third set of literature that we review is research on students who volunteer to 
participate in psychology experiments. These studies are useful given the study population, and 
because the decision to participate in these experiments is often similar to the decision to 
participate in a survey. 

The general population. In surveys of the general population, the most common finding 
is the impact of SES on response: more affluent or educated individuals are more likely to 
participate in surveys (Curtin, Presser, & Singer, 2000; Goyder, 1986; Goyder et ah, 2002; 
Groves et ah, 2000; Kandel & Ravels, 1983; Singer et ah, 1999; Singer, van Hoewyk, & Maher, 
2000) and are less likely to drop out of panel studies (Kandel & Ravels, 1983; Laurie, Smith, & 
Scott, 1999) than less affluent or educated individuals. Gender has also been found to play a 
significant role in survey response, with women responding in greater proportions than men 
(Curtin et ah, 2000; Groves & Couper, 1996; Moore & Tarnai, 2002; Singer et ah, 2000). Race 
is another demographic characteristic that has been linked to survey response (Curtin et ah, 2000; 
Groves et ah, 2000; Singer et ah, 1999; Singer et ah, 2000; Voigt et ah, 2003) and panel survey 
attrition (Kandel & Ravels, 1983), with whites responding more often than non-whites. 

Along with SES, race and gender, age is also correlated with survey response. Older 
individuals are less likely to participate in personal interviews (Lowe & McCormick, 1955) and 
surveys (Goyder, 1986; Kandel & Ravels, 1983; Moore & Tarnai, 2002). Using a sample of the 
elderly, Kaldenberg and his colleagues (Kaldenberg, Koenig, & Becker, 1994) found that non- 
response increased over one half of a percentage point for each year increase of respondents’ 



age. 
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Other research has compared how the relevance of the survey topic differs for 
respondents and non-respondents. Groves, Singer and Corning (2000) found that individuals 
with high levels of community involvement were more likely to respond to a survey regarding an 
issue being publicly debated at the time of the survey administration than individuals with low 
levels of community involvement. Similarly, to understand the relationship between survey 
topic involvement and survey participation, Kojetin, Borgida, and Snyder (1993) combined 
municipal recycling records with their survey data measuring recycling attitudes, behavior, and 
awareness and found that survey respondents recycled significantly more frequently than 
nonrespondents and that promptness of survey response was related to recycling attitudes, 
behaviors, and awareness. Individuals responding to the survey following the third contact (late 
responders) recycled less often and were less aware of and knowledgeable about of recycling 
than individuals responding after one of the first two contacts (Kojetin et ah, 1993). 

Further evidence of the relationship between topic involvement and survey participation 
comes from Van Kenhove, Wijnen, and De Wulf (2002), who compared two survey topics that 
differed on the average level of involvement in the sample. They found that differences in topic 
involvement between respondents and non-respondents only existed for the high involvement 
topic, suggesting that individuals’ involvement must surpass a critical level in order to influence 
the decision to participate in a survey. 

In addition to topic involvement, prior survey refusals and the number of previously 
completed surveys are two other behavioral characteristics that serve as important predictors of 
nonresponse (Brennan & Hoek, 1992; Goyder, 1986; Groves & Couper, 1996; Stinchcombe, 
Jones, & Sheatsley, 1981). As Bradburn (1978, p. 37) states. 
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On the whole, however, the fact that respondents have previously responded to an 
interview is the best predictor of subsequent participation, given that they can be 
located. After several waves of interviewing, one has probably gotten a sample of 
co-operative respondents who will continue to participate. 

College student populations. Research specific to student populations finds many of the 
same demographic correlates of survey response. For example, females usually respond at 
higher rates than males (Crawford et ah, 2001; Dey, 1997; Hutchinson, Tollefson, & Wigington, 
1987; National Center for Education Statistics, 2002; National Survey of Student Engagement, 
2003; Sax et ah, 2003) and White students are more likely to respond than non- whites (Dey, 
1997; National Center for Education Statistics, 2002). Academic factors are also related to 
survey participation. Dey (1997) found high school GPA and students’ self-ratings of academic 
ability to be the two student characteristics most correlated with survey response. Similarly, in a 
study equating late responders to non-respondents, late responders had lower GPAs than both 
early responders and the overall student population (Hutchinson et ah, 1987). Economic 
predictors of student response also mirror findings from the general public, with students on 
financial aid being more likely to respond than those not on aid (National Center for Education 
Statistics, 2002). However, because this study focused on financial aid, survey salience may also 
have contributed to the increased response by students on aid. 

The behavioral and psychological correlates of student nonresponse have also been 
explored. Johnson and Mowrer (2000) tested for personality differences between mail survey 
respondents and non-respondents. In their college student sample respondents did not differ 
from non-respondents on any of the “Big Eive” personality dimensions . While personality 

^ The “Big Five” personality dimension are Extraversion, Agreeableness, Conscientiousness, Neuroticsm, and 
Openness (McCrae & Costa, 1996). 
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differences have not emerged, there is some evidence that students’ level of engagement 
influences survey participation. Using telephone interviews to follow-up with NSSE non- 
responders, Kuh (2003) found that students who did not respond to the NSSE were slightly more 
engaged than their participating peers. Kuh cautions, however, that this finding may be 
attributed to the positive mode effects associated with the use of telephone interviews for follow 
up with NSSE non-respondents. 

Timing of subject pool participation. Einally, there is a body of research that does not 
look at student survey response, but nonetheless is very useful in understanding why some 
students take part in a survey. A substantial body of literature in psychology examines how 
students who volunteer for the research subject pool early in the semester differ from students 
who participate in the pool late in the semester. Demographic results from this research mirror 
the findings of the survey non-response literature. Eemales participate earlier in the semester 
than males (Cooper, Baumgardner, & Strathman, 1991; Evans & Donnerstein, 1974; Roman, 
Moskowitz, Stein, & Eisenberg, 1995; Stevens & Ash, 2001; Zelenski, Rusting, & Earsen, 2000) 
and students who participate earlier in the semester have higher GPAs (Aviv, Zelenski, Rallo, & 
Earsen, 2002; Cooper et ah, 1991; Evans & Donnerstein, 1974) and work fewer hours per week 
(Stevens & Ash, 2001) than students volunteering later in the semester. 

Psychologists have also tested whether motivational and personality characteristics are 
correlated with the timing of subject pool participation. While the results of these studies have 
been mixed"^ (Aviv et ah, 2002), studies focusing on the relationship between the “Big Eive” 
personality dimensions and subject pool participation have yielded some consistent findings. 
Eirst, an inverse relationship between extraversion and early participation has been replicated 
across studies (Aviv et ah, 2002; Eiberty, 1993). Additionally, Aviv and her colleagues (Aviv et 

^ This body of research has been inconsistent in both the constructs measured as well as the instruments employed. 
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al., 2002) and Stevens & Ash (2001) found a moderate negative correlation between week of 
participation and conscientiousness and a moderate positive relationship between week of 
participation and openness. In sum, this research suggests that late-semester participants are 
more extraverted, more open, and less conscientious than students volunteering early in the 
semester. 

Summary 

There are four main approaches to studying survey nonresponse. Nonresponse follow-up 
surveys and panel surveys can collect a rich variety of data, but are limited in that many 
nonrespondents are not included in the analysis. Studies examining the timing of subject pool 
participation can provide insight into how respondents and non-respondents differ, but the 
validity of this methodology may be called into question as non-respondents are not actually 
studied and because the point at which individuals are classified as ‘late responders’ is arbitrary. 
The record linkage approach avoids these biases, but is limited by the data available for analysis. 

Findings from these four approaches converge regarding the characteristics associated 
with survey participation: survey response is greatest for females. Whites, more affluent 
individuals, and those having higher levels of academic preparation, academic achievement, and 
engagement. Some studies have also linked attributes of personality to survey participation. If 
personality characteristics do indeed translate into survey response/non-response then there will 
be an additional source of error in our data, particularly for measures highly correlated with a 
dimension of personality (e.g., social engagement measures and extraversion). Furthermore, 
standard demographic -based weighting schemes may not account for these differences and our 



results will be biased. 
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Finally, the literature indicates that survey salience has a significant impact on survey 
participation. This research implies that any nonresponse study that relies on only one survey to 
study nonresponse may have biased results, as the impact of survey topic may vary across 
demographic and attitudinal groups. 



Methodology 

Our study combines the record linkage and panel survey approaches to study survey 
nonresponse at a selective liberal arts college. By combining these two approaches, we can rely 
on the strengths of each while minimizing their weaknesses. In our first set of analyses, we use 
the record linkage approach to obtain demographic and academic background information for all 
students from admission and course registration databases. Because all 462 students in the 
survey experiment are included, we have included typically noncooperative students in this part 
of the analysis. 

In our second set of analyses, we adopt the panel survey approach by using data from the 
CIRP freshman survey (see Sax, Lindholm, Astin, Korn, & Mahoney, 2001 for more 
information). Unlike many panel studies, the advantage of the CIRP at this particular institution 
is the high response rate, which for the three classes in this study range from 90-94%. Through 
an iterative merging process using SSN and demographic data, we can match 91% (N=420) of 
the full sample to their CIRP responses. While the missing 9% of students (N=42) may be 
atypical in that they are students who usually do not respond to surveys, this percentage is fairly 
low compared to many panel survey nonresponse studies. 

The result is two datasets available for analysis. The first contains all 462 students in the 



survey experiment, and the second contains the subset of 420 students who could be matched to 
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their CIRP survey responses. These students are all first-time, full-time degree-seeking students 
who were classified as sophomores, juniors, or seniors during the 2002-03 academic year.^ We 
focus on these 462 students because they were randomly selected from the student body to 
participate in a larger experiment measuring survey fatigue and were administered four different 
surveys as part of this larger experiment. 

Dependent variable 

Given our interest in understanding why students participate in surveys, how we 
construct the dependent variable is very important. Previous studies of the general population 
(Groves et ah, 2000; Kaldenberg et ah, 1994; Kojetin et ah, 1993) and college students (e.g., 
Kuh, 2003; Sax et ah, 2003) have focused on nonresponse in a single survey. The problem with 
this approach is that salience plays a large role in survey response, with response more likely 
when the survey topic is of interest to the respondent (Goyder, 1982; Groves et ah, 2000; 
Heberlein & Baumgartner, 1978; Tuckel & O'Neill, 2002). Survey salience can confound 
estimates of the impacts of independent variables; research indicates that the impact of 
demographic characteristics varies across survey topics (McDaniel, Madden, & Verille, 1987). 

To moderate the impact of salience we use a series of surveys conducted throughout the 
2002-03 academic year to measure survey cooperation. The surveys covered four different 
subjects: dining services, alcohol and drug use, student engagement behavior, and student 
satisfaction. All four surveys were administered via the web. Each survey had a response rate 



^ First-year students are excluded from our analysis because the CIRP administration method for this class changed 
from previous classes. The CIRP response rate for first-year students was 68%, too low to include them in the 
analysis. 




Survey participation 16 



for our experimental group between 38%-44%^. Rather than focus on a single survey, our 
dependent variable is the number of surveys in which a student participated, and ranges from 0 
(no surveys) to 4 (all surveys). The distribution of this variable is presented in Table 1 for the 
full sample. 

As can be seen in the first column, only 14% of students participated in all four surveys, 
even though the response rate for each individual survey was at least 38%. Fifty-seven percent 
participated in some but not all surveys, while 29% of the students were hard-core nonresponders 
and did not participate in any surveys. To put this last number in perspective, Tuckel and 
O’Neill (2002) found that nearly 40% of individuals in their study stated that they typically 
refused to participate in surveys, and Goyder (1986) found that almost half of the members of his 
sample refused their last contact requesting participation in an interview or mail survey. Thus it 
appears that in most populations a significant proportion of people never participate in a survey. 

To shed more light on how and when students respond. Table 1 shows the response 
profiles of the students. A ‘yes’ in a column indicates participation in that particular survey, and 
the table also shows the number and percentage of students in each response profile. As can be 
seen, students often chose to participate in some but not all surveys. Much of this variation is 
undoubtedly due to differences in survey salience. Couper (1997) corroborates this assertion, 
finding that individuals who either describe themselves as not interested in or as not 
knowledgeable about politics were less likely to take part in the 1990 National Election Study. 
The influence of survey content on participation (as well as other factors such as timing and 
survey length) shows the value of using several different surveys to measure survey cooperation. 



® Response rates in our experiment are Response Rate 6, as calculated by the American Association of Public 
Opinion Research (American Association for Public Opinion Research, 2004). This response rate assumes that there 
are no cases of unknown eligibility and it counts both partial and completed surveys as responses. 
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Independent variables 

We use five sets of independent variables to predict survey participation (see Table 3). 
The first consists of the standard demographic variables typical of many analyses of student 
outcomes: gender, racial/ethnic group, whether the student was on financial aid, and first 
generation college student. Given previous research we expect females to cooperate more often 
than males, whites more often than non-whites, and students from higher socio-economic 
backgrounds to cooperate more often. 

The second set measures the academic background of the student, their class year 
(sophomore or junior versus senior) and academic performance (end of year grade point average 
for the year the surveys were administered). Fewer senior students are expected to participate 
due to institutional survey fatigue, as they will have been contacted more times to participate in 
institutional surveys than students new to the institution. In line with previous research, we also 
expect students with high GPAs to be more likely to participate than their lower performing 
peers. However, the mechanism which leads students with superior academic credentials to 
participate in surveys is unknown. 

The third set of variables is a set of scales measuring pre-college engagement (see Table 
2). Engagement in college an important measure, as it is a major predictor of learning and 
personal development (e.g., Kuh, 2003). In addition, student engagement has been linked to 
survey response, with less engaged students possibly more likely to respond to surveys (Kuh, 
2003). We use two scales derived from the CIRP survey to measure student engagement: one 
measuring social engagement, the other measuring engagement in studying behavior. These 



variables are factor scores with a mean of 0 and a standard deviation of 1. 
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The fourth set of variables contains personality measures, also derived from the CIRP 
survey (see Table 2). These have been used to understand student behavior in other contexts 
(e.g., Smart, Feldman, & Ethington, 2000; Umbach & Porter, 2002). Personality type may affect 
survey response, as different types of student react to the various social norms and obligations 
that a request for survey participation evokes. Such norms and obligations play a key role in 
survey response (Groves, Cialdini, & Couper, 1992). As seen in Table 2, the four personality 
scales derived from the CIRP are investigative, social, artistic and enterprising (Smart et ah, 
2000). Since investigative personality types “value the development or acquisition of 
knowledge” (Smart et ah, 2000, p. 37), we expect investigative individuals to be survey 
cooperators, as they are likely to perceive our surveys as exercises aimed at acquiring 
information. These variables are factor scores with a mean of 0 and a standard deviation of 1. 

The final set of variables measure students’ survey behavior in the past. For the full 
sample, we use whether or not we were able to match a CIRP response as a measure of past 
survey participation behavior. Past survey participation has been shown to be a strong predictor 
of future survey participation (Goyder, 1986; Groves & Couper, 1996). While there is some 
measurement error in this variable due to the matching process, and also because some students 
may not have been present during orientation to take the survey, the magnitude of this variable 
will also allow us to understand whether the students not included in the second set of analyses 
differed in later survey participation than the included students. 

For the reduced sample, we include a variable indicating whether the student granted 
HERI permission to release their Social Security number back to the college in this study. This 
is an important variable because it acts as a proxy for privacy concerns and how the respondent’s 

^ “Do you give the Higher Education Research Institute (HERI) permission to include your ID number should your 
college request the data for additional research analyses? HERI maintains strict standards of confidentiality and 
would require your college to sign a pledge of confidentiality.” (Sax et ah, 2001). 
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data will be used. Previous research has indicated that attitudes about privacy in general as well 
as how it relates to survey research is correlated with survey response. 

Statistical technique 

We use an ordered logistic regression model to analyze survey response, as students had 
up to four opportunities to participate in a survey. This approach is useful when the dependent 
variable can be rank ordered, but the variable cannot be considered continuous (Long, 1997). 
For example, the difference between participating in 0 and 1 surveys may not be same as the 
difference between participating in 3 and 4 surveys. Ordered logit takes these possible 
nonlinearities into account. 



Results 

The ordinal logit results are presented in Table 3. Two sets of models were run: the first 
set on the full data created using the record linkage approach, and the second set on the reduced 
sample that combines the record linkage and panel study approaches. 

The full sample models use demographic and academic background data from 
institutional databases to predict survey response. As seen in the first column of the table, survey 
response is predicted by two demographic and one academic background measure. In line with 
previous research, females and students with high GPAs are more likely to take part in surveys, 
while students receiving financial aid are less likely to do so. 

Also using the full sample, we then added a variable that specified whether an 
individual’s data from the CIRP survey were available (see column 2). This dummy variable is 



coded as a ‘ 1 ’ in the cases where CIRP data were available and successfully merged with 
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institutional data and as a ‘0’ in the cases where CIRP data either did not exist or were not 
successfully merged with other measures. This variable treats individuals who did not 
participate in the CIRP survey and individuals for whom we could not successfully link CIRP 
data to other data identically. 

As seen in column 2, the availability of CIRP data is not a significant predictor of 
subsequent survey participation and therefore the omission of students without CIRP data from 
the subsequent models will not impact our overall results. Results from the second model using 
the full sample mirror results from the first model; the demographic and academic background 
measures found to be significant predictors of survey participation remain the same. Significant 
effects were found for females, students on financial aid and GPA, and the parameter estimates 
for these criteria are virtually identical to the estimates derived in the first model. 

Columns 3 and 4 of Table 3 present the results of analyses combining the record linkage 
and panel survey approaches. That is, these models include students for whom CIRP survey data 
were available and successfully linked to institutional database records. The model presented in 
column 3 replicates the analysis in column 1 in this new sample by testing the prediction of 
survey response solely from demographic and academic background variables. Results closely 
mirror the findings from the full sample model; again we see that females and students with high 
GPAs are more likely to participate in surveys, while students on financial aid are less likely to 
do so. 

In addition to demographic characteristics and academic background variables, the final 
model adds measures of engagement, personality type, and a dummy variable coding whether or 
not students granted permission to match their CIRP survey data to predict survey response (see 



column 4). 
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The effects of the demographic predictors remained unchanged: being female is 
positively related to survey response and receiving financial aid is negatively related to survey 
response. Student GPA is no longer a significant predictor of survey participation as in the 
previous models. This is likely due to the relationship between GPA and the personality scales; 
the measures of personality include questions about academic ability, and the personality scales 
now account for the variation previously attributed to GPA. 

The addition of CIRP panel survey data adds several significant predictors of survey 
response to the model. First there is a positive relationship between social engagement and 
survey participation, with socially engaged students being more likely to take part in surveys 
than their less engaged peers. This finding is in contrast to findings from the NSSE (Kuh, 2003). 
The difference in these findings is most likely due to the research design. Unlike the NSSE 
nonrespondent research, we have very little missing data in our research design. 

Personality measures derived from the CIRP survey were also found to be significant 
predictors of survey response. Individuals with high scores on the investigative personality type 
were more likely to respond to surveys than individuals scoring low on this trait. Conversely, 
higher scores for the artistic and enterprising personality measures predicted a decreased 
probability of survey participation. Note that the addition of the engagement and personality 
scales to the base model increases the pseudo R-squared by 43 percent. 

Eimitations 

The chief limitation of the current study is that we examine survey non-response at only 
one institution, a selective liberal arts college. Euture research should examine non-response 



patterns across multiple institutions of various types. 
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It is also unclear how generalizable our findings are to adult populations other than 
college students. The ability to replicate this research, however, is limited by the absence of data 
for the general population that can match the data available for our students. For example, 
information about individuals’ academic background, level of engagement, and personality are 
simply not available for the general public. 

While the record linkage approach provides many demographic and academic 
characteristics for studying survey response, the availability of panel survey data is limited. 
Scales were derived from the CIRP because this is the only pre-college survey with a high 
response rate that was available. Additional survey data would be useful to study survey 
nonresponse. Future studies, for example, could incorporate questions about student attitudes 
towards surveys, such as Goyder (1987). Only when we more fully understand the attitudes that 
dissuade students from participating in surveys, can we make targeted efforts to combat these 
drivers of survey nonresponse. 



Discussion 

By combining the record linkage and panel survey data approaches we were able to study 
survey nonresponse in a unique and informative way. This approach allowed us to go beyond 
the ubiquitous demographic and academic background predictors of survey participation, and 
examine student engagement, personality, and past survey behavior as they relate to survey 
response. Furthermore, by examining participation across multiple surveys we minimized the 



influence survey salience played on our dependent measure, survey response. 




Survey participation 23 



It is surprising that hard-core cooperators are such a small segment of population. Like 
Goyder (1986), we had speculated that a substantial proportion of students consistently 
participated in our surveys. This finding is somewhat reassuring in that it refutes the idea that 
there is one single group of students controlling the results of our surveying efforts. However, it 
is alarming that almost one-third of the students in our sample did not take part in any of the four 
surveys; our survey results do not include a substantial proportion of the student body. 

Our findings are in line with previous research regarding the role of gender, economic 
status (financial aid), and academic performance (GPA) in survey response. In addition to these 
standard demographic predictors, this study provides evidence that personality and student 
engagement impact survey participation, with more engaged students and students with 
investigate personalities more likely to respond, and students with enterprising personalities less 
likely to respond. 

To place these results in perspective. Table 4 compares the demographic, engagement, 
and personality characteristics of typical survey respondents to typical nonrespondents. We 
define typical respondents, or cooperators, as those students who responded to at least three of 
the four surveys and typical nonrespondents, or refusers, as those students who either did not 
respond to any surveys or responded to only one survey. This comparison provides us with a 
sense of what characteristics we might expect to see in the respondent and nonrespondent pools 
for a typical survey, regardless of topic. 

As can be seen, 66% of cooperators are female while only 42% of refusers are female. 
Over half of all refusers are on financial aid, while only 39% of cooperators received aid. These 
results indicate that many of our survey may be biased in favor of females and students with 
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higher socio-economic status. However, these results are less troubling, as most institutions can 
use institutional databases to weight responses based on gender and financial aid status. 

More troubling are the differences between cooperators and refusers for the engagement 
and personality scales. Social engagement, measured as a factor score, differs between refusers 
and cooperators by one-third of a standard deviation, with cooperators being more socially 
engaged than refusers. This finding suggests that surveys such as the NSSE may be 
overestimating engagement for schools, as more engaged students are more likely to participate 
in a survey. 

Substantive differences between cooperators and refusers also exist for two of the four 
personality types. Cooperators are more investigative and less enterprising than refusers, with 
the magnitude of these differences equaling one-third and one half of a standard deviation, 
respectively. This finding suggests that surveys geared towards specific topics, such as the 
natural sciences, may over (or under) estimate means and relationships between variables due to 
the self- selection of certain personality types. 

Using Holland’s personality typology as operationalized by Smart et al. (2000), we found 
evidence that student personality is related to survey response. Much of the previous research on 
personality and survey participation has focused on the relationship between the “Big Five” 
dimensions of personality, not Holland's typology. However, two meta-analytic studies (Barrick, 
Mount, & Gupta, 2003; Larson, Rottingham, & Borgen, 2002) have examined the 
intercorrelations of the “Big Five” and Holland's Big Six personality measures (For a review, see 
Mount, Barrick, Scullen, & Rounds, 2004). Of the 30 correlations, both studies found only four 
correlations greater than 0.25, and in both studies, these were for the same four pairs of 
measures. Of these four, the strongest correlations were between the enterprising type and 
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extraversion (Barrick et al., r = 0.41; Larson et al., r = 0.41) and the artistic type and openness 
(Barrick et al., r = 0.39; Larson et al., r = 0.48). 

Given the similarity between enterprising types and extraversion, our finding that 
enterprising types are likely to be survey refusers falls in line with research showing that 
extraverts tend to be latecomers to the subject pool (Aviv et al., 2002; Liberty, 1993). Similarly, 
given the strong correlation between the artistic type and openness, our finding that students with 
artistic personalities are less inclined to participate in our surveys replicates previous research 
showing that late-semester subject pool participants tend to have higher scores on the openness 
dimension than their early participating peers (Aviv et al., 2002; Stevens & Ash, 2001). 

Personality is emerging as an important factor in the decision to take part in surveys. By 
understanding the role of personality in survey participation more fully, we will be able to 
develop and employ techniques to counter survey nonresponse. Furthermore, since personality 
has also been shown to be a strong predictor of student major choice (Porter & Umbach, 2002), 
we must be aware that survey measures related to majors, for example satisfaction with a field of 
study or participation in laboratory courses, will be biased. Additionally, analysis comparing 
survey items across major fields of study will be prone the error. 

The current study also provides us with insight as to why response rates vary between 
institutions. Thes student composition across schools varies on many of the measures found to 
predict survey participation. For example, the socioeconomic makeup, or percentage of students 
receiving financial aid fluctuates from one school to another. The level of students’ academic 
preparation also varies across institutions. Finally, institutions themselves have “personalities” 
and social climates that lure certain types of individuals. Together, all of these factors promote 
the disparate response rates we see across institutions. 
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There are two main implications of these findings for research in higher education. First, 
we must discover effective ways to combat nonresponse among typically resistant demographic 
groups such as males. Second, we must consider ways to adjust our data for nonresponse in 
ways other than the standard demographic weighting schemes. Otherwise, measures such as 
student social engagement will be continue to be biased by the impact that student engagement 
and personality have on survey participation. 
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Table 1. Survey Participation 





Distribution in sample 

% N % 


N 


Alcohol & 
drug 


Surveys 

Engage- 

Dining ment 


Satisfaction 


Survey response rate 










41% 


45% 


38% 


39% 


Response profiles 












Responded to survey? 




All four surveys 


14.1 


65 


14.1 


65 


Yes 


Yes 


Yes 


Yes 


Three surveys 


16.0 


74 


2.2 


10 


Yes 


No 


Yes 


Yes 








2.6 


12 


Yes 


Yes 


No 


Yes 








5.8 


27 


Yes 


Yes 


Yes 


No 








5.4 


25 


No 


Yes 


Yes 


Yes 


Two surveys 


17.7 


82 


3.9 


18 


Yes 


Yes 


No 


No 








2.6 


12 


Yes 


No 


Yes 


No 








3.7 


17 


Yes 


No 


No 


Yes 








3.0 


14 


No 


Yes 


Yes 


No 








3.2 


15 


No 


Yes 


No 


Yes 








1.3 


6 


No 


No 


Yes 


Yes 


One survey 


23.2 


107 


6.3 


29 


Yes 


No 


No 


No 








6.5 


30 


No 


Yes 


No 


No 








3.7 


17 


No 


No 


Yes 


No 








6.7 


31 


No 


No 


No 


Yes 


No surveys 


29.0 


134 


29.0 


134 


No 


No 


No 


No 


Total 


100.0 


462 


100.0 


462 
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Table 2. Scales 

Scale Alpha CIRP item 

Engagement: 0.72 Frequency in high school: performed volunteer work 

social Frequency in high school: discussed religion 

Frequency in high school: discussed politics 

Frequency in high school: attended a public recital or concert 

Frequency in high school: visited an art gallery or museum 

Frequency in high school: participated in organized demonstrations 

Frequency in high school: voted in a student election 

Hours in high school: student clubs/groups 

Hours in high school: volunteer work 

Plans for college: participate in volunteer or community service work 
Plans for college: participate in student protests or demonstrations 
Plans for college: partcipate in student government 

Engagement: 0.61 Frequency in high school: studied with other students 

studying Frequency in high school: was a guest in a teacher’s home 

Frequency in high school: asked a teacher for advice after class 

Hours in high school: studying/homework 

Hours in high school: talking with teachers outside of class 

Personality: 0.57 Goal: making a theoretical contribution to science 

investigative Rating: academic 

Rating: drive to achieve 
Rating: mathematical ability 
Rating: self-confidence (intellectual) 

Personality: 0.80 Goal: becoming involved in programs to clean up the environment 

social Goal: helping others who are in difficulty 

Goal: helping to promote racial understanding 

Goal: influencing social values 

Goal: influencing the political structure 

Goal: participating in a community action program 

Personality: 0.73 Goal: becoming accomplished in one of the performing arts 

artistic Goal: creating artistic work 

Goal: developing a meaningful philosophy of life 
Goal: writing original works 
Rating: artistic ability 
Rating: writing ability 

Personality: 0.74 Goal: becoming an authority in my field 

enterprising Goal: becoming successful in a business of my own 

Goal: being very well off financially 

Goal: having administrative responsibility for the work of others 

Goal: obtaining recognition from my colleagues for contributions to my field 

Rating: leadership ability 

Rating: popularity 

Rating: self-confidence (social) 
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Table 3. Model Results 





Pull sample 




CIRP respondents only 




1 


2 


3 


4 


Demographics 










Female 


0.707 ** 


0.710 ** 


0.819 ** 


0.794 ** 


African 


-0.442 


-0.418 


-0.597 + 


-0.545 


Asian 


0.010 


0.018 


-0.118 


0.069 


Hispanic 


-0.520 


-0.530 


-0.516 


-0.408 


Other/unknown 


0.270 


0.295 


0.262 


0.431 


Non-resident alien 


-0.242 


-0.256 


-0.377 


-0.083 


First generation 


0.172 


0.181 


0.195 


0.229 


Financial aid status 


-0.408 * 


-0.420 * 


-0.413 * 


-0.432 * 


Academic background 










Class - sophomore 


0.203 


0.205 


0.253 


0.311 


Class - junior 


-0.353 


-0.365 + 


-0.337 


-0.212 


Cumulative GPA 


0.078 ** 


0.078 ** 


0.070 ** 


0.041 


Engagement scales 










Social 








0.350 ** 


Studying 
Personality scales 








0.068 


Investigative 








0.300 ** 


Artistic 








-0.198 * 


Social 








-0.151 


Enterprising 
Past survey behavior 








-0.319 ** 


CIRP nonmatch 
CIRP ID refusal 




-0.215 




-0.017 


Cutpoints 










I 


-8.896 ** 


-8.871 ** 


-8.243 ** 


-5.888 * 


2 


-7.847 ** 


-7.820 ** 


-7.172 ** 


-4.782 * 


3 


-7.006 ** 


-6.978 ** 


-6.355 ** 


-3.924 


4 


-5.930 ** 


-5.902 ** 


-5.274 * 


-2.789 


N 


462 


462 


420 


420 


Pseudo R-square 


0.14 


0.14 


0.14 


0.20 


-2 log likelihood 


1390.3 


1389.7 


1260.3 


1233.4 



Note: ** p<.01, * p<.05, + p<.10. 
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Table 4. Differences Between Refusers and Cooperators 



Variables 


Refusers 
(responded to 
0 or 1 surveys) 


Cooperators 
(responded to 
3 or 4 surveys) 


Difference 


Demographics 


Female (%) 


41.6 


65.9 


24.4 


On financial aid (%) 


58.0 


38.8 


-19.2 


Engagement scale 


Social 


-0.14 


0.19 


0.33 


Personality scales 


Investigative 


-0.21 


0.08 


0.29 


Artistic 


-0.22 


-0.14 


0.08 


Social 


-0.02 


0.06 


0.08 


Enterprising 


0.16 


-0.31 


-0.47 
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Research approach 

Time of response analysis 

Respondents to the base survey are analyzed based on when 
they completed the survey, early in the survey process 
versus late in the survey process. These two groups are 
compared, with no additional data collection. 



Nonresponse followup survey 

Nonrespondents (or a subset of nonrespondents) to the base 
survey are administered a second survey, usually with 
intense efforts to ensure response. Respondents to the 
followup survey are compared with original respondents to 
the base survey, using data from both surveys. 



Panel survey 

Respondents to the base survey are administered one or 
more followup survey. Nonrespondents in these successive 
surveys are compared to respondents, with nonresponse 
analyzed using data from the base survey. 



Record linkage 

Both respondents and nonrespondents to the base survey are 
linked to database records that are available for the full 
sample, such as demographic or financial information. 

These data are used to analyze nonresponse. 



□ 

□n 



information is available for this group 
almost nothing is known about this group 



groups compared to understand nonresponse 



Base survey Additional data 
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